{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hill Climbing (with adaptive noise scaling, without tensors)\n",
    "# CartPole-v0\n",
    "\n",
    "---\n",
    "\n",
    "   In this notebook, we train the Hill Climbing Agent with   \n",
    "**adaptive noise scaling** with OpenAI Gym's Cartpole environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy-Based Method\n",
    "\n",
    "In **policy-based methods**, instead of learning a **value function** that tells us what is     \n",
    "the expected sum of rewards given a state and an action, we learn directly the **policy function**    \n",
    "that maps state to action (select actions without using a value function)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "#%matplotlib inline\n",
    "import time\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define the Policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "observation space: Box(4,)\n",
      "action space: Discrete(2)\n",
      "threshold:  195.0\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "print('observation space:', env.observation_space)\n",
    "print('action space:', env.action_space)\n",
    "threshold = env.spec.reward_threshold\n",
    "print('threshold: ', threshold)\n",
    "\n",
    "class Policy():\n",
    "    def __init__(self, s_size=4, a_size=2):\n",
    "        self.w = 1e-4*np.random.rand(s_size, a_size)  # weights for simple linear policy: state_space x action_space\n",
    "        \n",
    "    def forward(self, state):\n",
    "        x = np.dot(state, self.w)\n",
    "        return np.exp(x)/sum(np.exp(x))\n",
    "    \n",
    "    def act(self, state):\n",
    "        probs = self.forward(state)\n",
    "        #action = np.random.choice(2, p=probs) # option 1: stochastic policy\n",
    "        action = np.argmax(probs)              # option 2: deterministic policy\n",
    "        return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with Stochastic Policy Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep.:   1 , timesteps: 192, noise_scale (Gt >= best(Gt)): 0.0050, Gt 85.4803, \tAvg.Score:  192.000\n",
      "Ep.:   2 , timesteps:  54, noise_scale (Gt < best(Gt)): 0.0100, Gt 41.8834, \tAvg.Score:  123.000\n",
      "Ep.:   3 , timesteps:  82, noise_scale (Gt < best(Gt)): 0.0200, Gt 56.1382, \tAvg.Score:  109.333\n",
      "Ep.:   4 , timesteps:  11, noise_scale (Gt < best(Gt)): 0.0400, Gt 10.4662, \tAvg.Score:  84.750\n",
      "Ep.:   5 , timesteps:   8, noise_scale (Gt < best(Gt)): 0.0800, Gt 7.7255, \tAvg.Score:  69.400\n",
      "Ep.:   6 , timesteps:  32, noise_scale (Gt < best(Gt)): 0.1600, Gt 27.5020, \tAvg.Score:  63.167\n",
      "Ep.:   7 , timesteps:  10, noise_scale (Gt < best(Gt)): 0.3200, Gt 9.5618, \tAvg.Score:  55.571\n",
      "Ep.:   8 , timesteps:   9, noise_scale (Gt < best(Gt)): 0.6400, Gt 8.6483, \tAvg.Score:  49.750\n",
      "Ep.:   9 , timesteps:  10, noise_scale (Gt < best(Gt)): 1.2800, Gt 9.5618, \tAvg.Score:  45.333\n",
      "Ep.:  10 , timesteps:  34, noise_scale (Gt < best(Gt)): 2.0000, Gt 28.9447, \tAvg.Score:  44.200\n",
      "Ep.:  11 , timesteps:   8, noise_scale (Gt < best(Gt)): 2.0000, Gt 7.7255, \tAvg.Score:  40.909\n",
      "Ep.:  12 , timesteps:  11, noise_scale (Gt < best(Gt)): 2.0000, Gt 10.4662, \tAvg.Score:  38.417\n",
      "Ep.:  13 , timesteps:  42, noise_scale (Gt < best(Gt)): 2.0000, Gt 34.4341, \tAvg.Score:  38.692\n",
      "Ep.:  14 , timesteps: 200, noise_scale (Gt >= best(Gt)): 1.0000, Gt 86.6020, \tAvg.Score:  50.214\n",
      "Ep.:  15 , timesteps:  95, noise_scale (Gt < best(Gt)): 2.0000, Gt 61.5104, \tAvg.Score:  53.200\n",
      "Ep.:  16 , timesteps: 103, noise_scale (Gt < best(Gt)): 2.0000, Gt 64.4839, \tAvg.Score:  56.312\n",
      "Ep.:  17 , timesteps: 155, noise_scale (Gt < best(Gt)): 2.0000, Gt 78.9402, \tAvg.Score:  62.118\n",
      "Ep.:  18 , timesteps:  68, noise_scale (Gt < best(Gt)): 2.0000, Gt 49.5114, \tAvg.Score:  62.444\n",
      "Ep.:  19 , timesteps: 200, noise_scale (Gt >= best(Gt)): 1.0000, Gt 86.6020, \tAvg.Score:  69.684\n",
      "Ep.:  20 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.5000, Gt 86.6020, \tAvg.Score:  76.200\n",
      "Ep.:  21 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.2500, Gt 86.6020, \tAvg.Score:  82.095\n",
      "Ep.:  22 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.1250, Gt 86.6020, \tAvg.Score:  87.455\n",
      "Ep.:  23 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0625, Gt 86.6020, \tAvg.Score:  92.348\n",
      "Ep.:  24 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0312, Gt 86.6020, \tAvg.Score:  96.833\n",
      "Ep.:  25 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0156, Gt 86.6020, \tAvg.Score:  100.960\n",
      "Ep.:  26 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0078, Gt 86.6020, \tAvg.Score:  104.769\n",
      "Ep.:  27 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0039, Gt 86.6020, \tAvg.Score:  108.296\n",
      "Ep.:  28 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0020, Gt 86.6020, \tAvg.Score:  111.571\n",
      "Ep.:  29 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  114.621\n",
      "Ep.:  30 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  117.467\n",
      "Ep.:  31 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  120.129\n",
      "Ep.:  32 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  122.625\n",
      "Ep.:  33 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  124.970\n",
      "Ep.:  34 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  127.176\n",
      "Ep.:  35 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  129.257\n",
      "Ep.:  36 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  131.222\n",
      "Ep.:  37 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  133.081\n",
      "Ep.:  38 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  134.842\n",
      "Ep.:  39 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  136.513\n",
      "Ep.:  40 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  138.100\n",
      "Ep.:  41 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  139.610\n",
      "Ep.:  42 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  141.048\n",
      "Ep.:  43 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  142.419\n",
      "Ep.:  44 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  143.727\n",
      "Ep.:  45 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  144.978\n",
      "Ep.:  46 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  146.174\n",
      "Ep.:  47 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  147.319\n",
      "Ep.:  48 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  148.417\n",
      "Ep.:  49 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  149.469\n",
      "Ep.:  50 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  150.480\n",
      "Ep.:  51 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  151.451\n",
      "Ep.:  52 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  152.385\n",
      "Ep.:  53 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  153.283\n",
      "Ep.:  54 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  154.148\n",
      "Ep.:  55 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  154.982\n",
      "Ep.:  56 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  155.786\n",
      "Ep.:  57 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  156.561\n",
      "Ep.:  58 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  157.310\n",
      "Ep.:  59 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  158.034\n",
      "Ep.:  60 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  158.733\n",
      "Ep.:  61 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  159.410\n",
      "Ep.:  62 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  160.065\n",
      "Ep.:  63 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  160.698\n",
      "Ep.:  64 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  161.312\n",
      "Ep.:  65 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  161.908\n",
      "Ep.:  66 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  162.485\n",
      "Ep.:  67 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  163.045\n",
      "Ep.:  68 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  163.588\n",
      "Ep.:  69 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  164.116\n",
      "Ep.:  70 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  164.629\n",
      "Ep.:  71 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  165.127\n",
      "Ep.:  72 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  165.611\n",
      "Ep.:  73 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  166.082\n",
      "Ep.:  74 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  166.541\n",
      "Ep.:  75 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  166.987\n",
      "Ep.:  76 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  167.421\n",
      "Ep.:  77 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  167.844\n",
      "Ep.:  78 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  168.256\n",
      "Ep.:  79 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  168.658\n",
      "Ep.:  80 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  169.050\n",
      "Ep.:  81 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  169.432\n",
      "Ep.:  82 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  169.805\n",
      "Ep.:  83 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  170.169\n",
      "Ep.:  84 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  170.524\n",
      "Ep.:  85 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  170.871\n",
      "Ep.:  86 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  171.209\n",
      "Ep.:  87 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  171.540\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep.:  88 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  171.864\n",
      "Ep.:  89 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  172.180\n",
      "Ep.:  90 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  172.489\n",
      "Ep.:  91 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  172.791\n",
      "Ep.:  92 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  173.087\n",
      "Ep.:  93 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  173.376\n",
      "Ep.:  94 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  173.660\n",
      "Ep.:  95 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  173.937\n",
      "Ep.:  96 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  174.208\n",
      "Ep.:  97 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  174.474\n",
      "Ep.:  98 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  174.735\n",
      "Ep.:  99 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  174.990\n",
      "Ep.: 100 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  175.240\n",
      "Ep.: 101 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  175.320\n",
      "Ep.: 102 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  176.780\n",
      "Ep.: 103 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  177.960\n",
      "Ep.: 104 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  179.850\n",
      "Ep.: 105 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  181.770\n",
      "Ep.: 106 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  183.450\n",
      "Ep.: 107 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  185.350\n",
      "Ep.: 108 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  187.260\n",
      "Ep.: 109 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  189.160\n",
      "Ep.: 110 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  190.820\n",
      "Ep.: 111 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  192.740\n",
      "Ep.: 112 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  194.630\n",
      "Ep.: 113 , timesteps: 200, noise_scale (Gt >= best(Gt)): 0.0010, Gt 86.6020, \tAvg.Score:  196.210\n",
      "Environment solved in 113 episodes!\tAverage Score: 196.21\n"
     ]
    }
   ],
   "source": [
    "env.seed(0)\n",
    "np.random.seed(0)\n",
    "\n",
    "policy = Policy()\n",
    "\n",
    "def hill_climbing(n_episodes=1000, gamma=0.99, print_every=100, noise_scale=1e-2):\n",
    "    \"\"\"Implementation of hill climbing with adaptive noise scaling.\n",
    "        \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        gamma (float): discount rate\n",
    "        print_every (int): how often to print average score (over last 100 episodes)\n",
    "        noise_scale (float): standard deviation of additive noise\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores = []\n",
    "    arr_noise = []\n",
    "    best_Gt = -np.Inf\n",
    "    best_w = policy.w\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        rewards = []\n",
    "        state = env.reset()\n",
    "        timesteps = 0 ## is the same as len(rewards)\n",
    "        \n",
    "        while True:\n",
    "            action = policy.act(state)\n",
    "            state, reward, done, _ = env.step(action)            \n",
    "            rewards.append(reward)\n",
    "            timesteps += 1  \n",
    "            if done:\n",
    "                break \n",
    "                \n",
    "        total_reward = sum(rewards)        \n",
    "        scores_deque.append(total_reward)\n",
    "        scores.append(total_reward)\n",
    "\n",
    "        discounts = [gamma**i for i in range(len(rewards)+1)]\n",
    "        ## This is the 'cumulative discounted reward' or TD-target G_t\n",
    "        Gt = sum([a*b for a,b in zip(discounts, rewards)])\n",
    "        \n",
    "        if Gt >= best_Gt: # found better weights\n",
    "            ## if Gt > best_R ==> decrease the noise: noise = noise/2  (till 0.001)\n",
    "            best_Gt = Gt\n",
    "            best_w = policy.w\n",
    "            noise_scale = max(1e-3, noise_scale / 2)\n",
    "            print('Ep.: {:3d} , timesteps: {:3d}, noise_scale (Gt >= best(Gt)): {:.4f}, Gt {:.4f}, \\tAvg.Score:  {:.3f}'\\\n",
    "                  .format(i_episode, timesteps, noise_scale, Gt, np.mean(scores_deque)))\n",
    "            policy.w += noise_scale * np.random.rand(*policy.w.shape) \n",
    "        else: # did not find better weights\n",
    "            ## if Gt < best_R ==> increase the noise: noise = 2*noise (till 2)\n",
    "            noise_scale = min(2, noise_scale * 2)\n",
    "            print('Ep.: {:3d} , timesteps: {:3d}, noise_scale (Gt < best(Gt)): {:.4f}, Gt {:.4f}, \\tAvg.Score:  {:.3f}'\\\n",
    "                  .format(i_episode, timesteps, noise_scale, Gt, np.mean(scores_deque)))\n",
    "            policy.w = best_w + noise_scale * np.random.rand(*policy.w.shape)\n",
    "            \n",
    "        arr_noise.append(noise_scale)   \n",
    "\n",
    "        if np.mean(scores_deque)>=threshold:\n",
    "            print('Environment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            policy.w = best_w\n",
    "            break\n",
    "    \n",
    "    return scores, arr_noise\n",
    "            \n",
    "scores, arr_noise = hill_climbing()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Plot the Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5xkdX3n/9e7Ln2ZGeYCMwPDAA4oosQYMK1xV2VBTaKuqzGbRFkjxLgSo0bdmE3U/f1Wze+X3yYmauIvCZFEImYNYsQLSVhXQlwxGyHOCBm534I6MMz0MDAwt+6uqs/+cc6pPt1d3VPVU6equvv9fDzq0VWnbt+agvOpz/fzvSgiMDMzAyj1uwFmZjY4HBTMzKzJQcHMzJocFMzMrMlBwczMmir9bsDx2LhxY2zbtq3fzTAzW1J27NixLyI2tbpvSQeFbdu2sX379n43w8xsSZH0vfnuc/eRmZk1OSiYmVmTg4KZmTU5KJiZWZODgpmZNRUWFCSdLunrku6SdIekd6fHT5R0g6T70r8b0uOS9AlJ90vaKel5RbXNzMxaKzJTqAHvjYhnAy8E3iHpXOB9wI0RcTZwY3ob4JXA2enlMuDyAttmZmYtFDZPISJ2A7vT609JugvYCrwWuDB92FXA/wJ+Iz3+mUjW8r5Z0npJW9LXWdb+8f59nLJuhLM2rWn7OYcna3z6Hx/i6GS9wJaZ2aB65ikn8Ornntr11+3J5DVJ24DzgVuAk7MTfUTslrQ5fdhW4Ae5p+1Kj80ICpIuI8kkOOOMMwptd6/8+rU7+ddPP4mP/MyPtP2cWx7cz0e+eg8AUlEtM7NB9ernnro0g4KkNcC1wHsi4knNfwZrdcecHYAi4grgCoCxsbFlsUPQRK3BkalGh89JMoTr3/USzj11bRHNMrMVqNDRR5KqJAHhsxHxxfTwHklb0vu3AHvT47uA03NPPw14pMj2DYp6I5iqdRYUJutJPKyWnSaYWfcUOfpIwKeAuyLiY7m7rgMuTa9fCnwld/ySdBTSC4EDK6GeADBVbzBZ7ywo1NLHV8oeVWxm3VNk99GLgDcB35V0W3rsA8BvA5+X9Bbg+8DPpvddD7wKuB84DLy5wLYNlHojmOo4KCSZQqXkTMHMuqfI0Uf/QOs6AcDLWjw+gHcU1Z5BVmsEEx12H001ksdXnSmYWRf5jDIAjitTcE3BzLpoRQaFJ49OcdO94zx2cKLfTSEiqDeCyU4zhTSIVEsr8is0s4KsyDPKg+OHuOTKf+Kfdz3R76ZQayS/+DvOFBrOFMys+1ZkUMiKs1P1/k9zqDeDQmdtyUYfuaZgZt20Is8o2Ym0NgBBIcsQOu8+8jwFM+u+FRkUymmmUGt0diIuQpYpdDpPYareoFwSC8wQNzPr2IoMCtmv60HIFLLaQKeZQq0RnqNgZl23IoNCNgt4kDKFTgvNU/WG6wlm1nUr8qxSHaBC82JrCrV6eOSRmXXdigwKzUyhw1/nRcgyhVojaDTaD1K1RoOK5yiYWZetyLNK9gu71sFJuCj5Nkx10J01VQ+PPDKzrluRQSGbBTwI3Uf5YncnXUg11xTMrAAr8qzSzBQGoPsoX+zuJEhNNVxTMLPuW5lBISs0D0D3Ub2xuExhqtbwukdm1nUr8qwiiUpJA5Ip5GoKHbSn5kzBzAqwIoMCJLOaB6LQnOsy6mRPhal6w7uumVnXrdizSrVcGpAZzfmaQieF5mjOtzAz65Yi92i+UtJeSbfnjl0j6bb08lC2TaekbZKO5O77k6LalamUNVAzmqHD0UeNhruPzKzrityj+dPAHwKfyQ5ExOuz65I+ChzIPf6BiDivwPbMUCmVBmNI6iJrClP1YKRaLqJJZraCFblH802StrW6T8nSnj8HvLSo9z+WanlACs35eQodFZo9T8HMuq9fZ5WXAHsi4r7csTMl3SrpG5JeMt8TJV0mabuk7ePj44tuQNJ91P9MoZ7rwups8ppXSTWz7utXULgYuDp3ezdwRkScD/wq8JeS1rZ6YkRcERFjETG2adOmRTegWip1vDJpEWZ2H3Uwec0zms2sAD0/q0iqAD8NXJMdi4iJiHgsvb4DeAB4ZpHtqJQ1GKOPFrnMxZRXSTWzAvTjp+bLgbsjYld2QNImSeX0+lnA2cCDRTaiUioNxOijRU9eq3uVVDPrviKHpF4NfAs4R9IuSW9J73oDM7uOAC4Adkr6Z+ALwNsiYn9RbYOk0DwIo48WW1OYaniVVDPrviJHH108z/FfaHHsWuDaotrSSjKjebAyhY5GH9U9T8HMum/F9j9UBmVGc32x3Ufh7iMz67oVe1apDsiQ1NpiV0ltNNx9ZGZdt2KDQqVUGojJa/XjWfvIQ1LNrMtW7FllUArNU4sYkhoR6dLZK/brM7OCrNizyqAMSc0WxCuXxGSbQSoLJF4l1cy6beUGhUGZvJYGhdFque1MIQtmzhTMrNtW7FmlWi4xNRCZQoNKSQxV2l92o5kpuNBsZl22YoNCsh3nAGQK9aBcEtWy2s8U0uDhBfHMrNtWblAoD85+Cp1mClmXk7uPzKzbVuxZpTIgM5rrjSxTKLU9ozkLHu4+MrNuW7lBoSzqA5ApZEtgD5VLHXQfZTWFFfv1mVlBVuxZZXAKzUmm0Fn3kUcfmVkxVuxZZWAKzWlNobPuI89TMLNirNygUC5RawQR/Q0M9XRm8lC5xFStdVuOTtU5OlVv3s4yCmcKZtZtK/askv3K7veieFP1ZJ5CtVJiYp5M4Teu3cmvXH1r7jnZ6CNnCmbWXYXtpzDosl/ZycJy/WtHs6ZQLjE1T6H5e48dnjFSKpunUPXS2WbWZSs2KGTDOacaDUbpX1SoNQvNmrfQfGiiNuc54EzBzLqvyO04r5S0V9LtuWMfkvSwpNvSy6ty971f0v2S7pH0k0W1K5PNBu53sbneiOkhqQsEhYna3CW2PU/BzLpt3qAgaY2k35R0h6QDksYl3SzpF9p87U8Dr2hx/OMRcV56uT59r3NJ9m7+ofQ5fyyp0J/v091H/R2WOlVvNCevzdd9dGiyPmMOg+cpmFlRFjqrfBZ4EPhJ4MPAJ4A3ARdJ+v+O9cIRcROwv812vBb4XERMRMS/APcDL2jzuYuSZQpTfS4017MhqZXWmUJEpJnC9Oij5jwF1xTMrMsWOqtsi4hPR8SuiPgY8JqIuA94M/DTx/Ge75S0M+1e2pAe2wr8IPeYXemxOSRdJmm7pO3j4+OLbkSWKfR7VnOyWY7mndE8UWtQa8Ss7iOvkmpmxVgoKByS9GIASf+O9Fd/RDSAxZ6NLgeeDpwH7AY+mh5v9Xotz9YRcUVEjEXE2KZNmxbZjJmF5n6q1RtUSiWG5skUsiJzq5qC5ymYWbctNProbcCfSToH+C7wFgBJm4A/WsybRcSe7LqkPwX+Jr25Czg999DTgEcW8x7tyrpeBqHQ3ByS2qIthybqzcfV6o1k0l02T8Ezms2sy+b9qRkROyPiBRGxLiJeHBH3pMfHI+ITi3kzSVtyN18HZCOTrgPeIGlY0pnA2cA/LeY92pUN52x3vaGi5Je5qDeiuT1n5mBuOGqWSWTZjQvNZtZtC85TkPR0kpP36UANuA+4OiIOHOuFJV0NXAhslLQL+CBwoaTzSLqGHgJ+CSAi7pD0eeDO9H3eERH1Vq/bLVn3Ub9nNNezmkIlOcEno5GmB14dmpwOChNTDVYNTWc3nqdgZt02b1CQ9C7g3wHfAJ4P3EYSHL4l6e0R8b8WeuGIuLjF4U8t8PjfAn6rjTZ3xXT3Uf+HpFZKpWaQmqw3GMlNsc5nClldYcozms2sIAtlCm8FzouIuqSPAddHxIWSPgl8BTi/Jy0syHT3Uf8zhWzpbGDOCKT8bObsPs9oNrOiHOunZhY0hoETACLi+0C1yEb1QtYf3+/d1/JDUmFujePQjEwh6VFrrn3kmoKZddlCmcKfAd+WdDNwAfA70Bx91O6ktIE1KMtc1OrThWaYmykcnJgurUx3H3megpkVY96gEBF/IOnvgGcDH4uIu9Pj4yRBYkmrzvPLvNeSBfFKMwrNeS0zhUayNIbkoGBm3bXg6KOIuAO4o0dt6alyminMHgLaa/VGY0amMLFATSGfKXiOgpkVYVGd0pL+5tiPGmzTM5oHY5mL4WamMP88hfzoI9cTzKwIiz2zvLWrreiDQRmSOrumsGD30VRj+jmuJ5hZAdoKCpJOzC1eR0TsLq5JvZGdVPtdaK6nNYXmPIUWheas3pCvKXiFVDMrwkL7KZwh6XOSxoFbSEYi7U2PbetVA4vS/GXe9yGpSU2hOU+hRaZw0uqh5L5cTcEjj8ysCAv93LwG+BJwSkScHRHPALYAXwY+14vGFWkQhqQ2GkEjkqxlviGphyZrbFiVBIWsplBzTcHMCrLQmWVjRFyTX4MoIuoR8TngpOKbVqzKAAxJbc5MLuULzbO7j2qctGZmUJhquKZgZsVYaEjqDkl/DFzF9AY4pwOXArcW3bCiDcKCeNlw2KSmMP8yFyeuzoLC9Ixmr3tkZkVYKChcQrKHwodJdkETyb4H17HAwnZLxSCMPqo1l8DWApPX6s3uo3xNwZmCmRVhoRnNkyQ7pV3eu+b0TnOP5j7WFLJ6Rjm/zEWuPRHBockaJ4xUGCqXZsxT8K5rZlaEdoek/nz+73JQKomS+jujOV9TaLVK6uHJOhGwerjCcKU0Y55C1TOazawA7f7c/NVZf5eFSrnU1yGp+ZpCq1VSs4lrq4crDFdLM+cpuPvIzArQaR9E22ciSVem8xpuzx37XUl3S9op6UuS1qfHt0k6Ium29PInHbZrUaol9XVIalZTSIakzp28li1xsWa4PKv7KDwk1cwKUeSZ5dPAK2YduwF4TkQ8F7gXeH/uvgci4rz08rYC29VUKZf6W2iuT3cfVcolSpqdKSSZweqhCsPVcm6THc9TMLNiFHZmiYibmLXvQkR8LSKyxXxuBk4r6v3bUS2rrwvi1RrTheakPaV5MoW0ptAckupVUs2sGP38ufmLwP/I3T5T0q2SviHpJfM9SdJlkrZL2j4+Pn5cDaiU+pspZDWF7Ff/UKU0Y5mLGTWFSsmrpJpZ4do9s9yb/r2nG28q6b8ANeCz6aHdwBkRcT5JMfsvJa1t9dyIuCIixiJibNOmTcfVjkq5vzWFrKsoyxSGZmUKhybzQaE8PfrIM5rNrCBtBYWIeEP+7/GQdCnwauCNERHp605ExGPp9R3AA8Azj/e9jqVaLvW1+6ieG5IKSaaQrynku4/yWcRUzaukmlkxFtx5LZMum302MJIdS2sGHZH0CuA3gH8TEYdzxzcB+yOiLums9L0e7PT1O1Upqc8zmufWFPKT6aa7j8oMV0o8cWQSSNY+8iqpZlaEYwYFSf8ReDdJUfg24IXAt4CXHuN5VwMXAhsl7QI+SDLaaBi4Id1f+OZ0pNEFwG9KqgF14G0Rsb/lC3dRZdZJuNdm1xSqZc0qNOdHH+Unr3megpkVo51M4d3A80lO4BdJehbJekgLioiLWxxuuWZSRFwLXNtGW7qqUhL1Pk5eq82uKVTKcwrNq4bKlEpKagq16RnN7j4ysyK0c2Y5GhFHASQNR8TdwDnFNqs3KmX1dZXU2uyawqxM4dBEjdXDlfS+6SL0VKPh7iMzK0Q7mcKudObxl0m6fR4HHim2Wb1RLZX6up9Cs9CcG5I6u9C8Jg0KM5a58IxmMyvIMYNCRLwuvfohSV8H1gFfLbRVPVKZ9cu817IAUJln8lqSKZQBmvMUIiIdkuqgYGbd1+4qqRskPRd4imRPhecU2qoeqQzIkNTyPENSD03UWT2UZgppTSHrcvIqqWZWhHZGH/0/wC+QDBHNzljBMUYfLQXVARmSOiNTyI2GOjhR49T1ySjgoUqJeiM4OpV0ITlTMLMitFNT+Dng6emmO8tKv2c0z6kplEtM1ppbYnNocrrQnO3hnC2S50KzmRWhnZ+btwPri25IP/R7P4XZNYWk+2jm5LXZQSGb5ewF8cysCO1kCv8NuDXdF2EiOxgRrymsVT3S7/0UZtcU5k5ey48+SgrOh9P1kNx9ZGZFaCcoXAX8DvBdpmsKy0Lf91Nodh/NLTTX6g2OTjVyheaZmYK7j8ysCO0EhX0R8YnCW9IHlVJ/J69NL4iXLXMxPST10GS6xEU6JHVoTk3BmYKZdV87QWGHpP8GXMfM7qPvFNaqHun3jOY5S2fnVkI9lFshFZIhqfnj7j4ysyK0ExTOT/++MHdsWQxJrQzKjOb8fgr1ZIJafoMdaNF95EKzmRWgnRnNF/WiIf1Q7fOQ1Dk1hXKJiCRYHJyTKWTdR84UzKw47UxeWw9cAmzLPz4i3lVcs3qjUi5R6+sqqbNqCumJf6oezdpBc0G8OUHBmYKZdV873UfXAzezDEcfVUtiqh5EBOn+Dj2VLdud9QRlxePJWqOZKUyvfZT8zfZYqHrpbDMrQDtBYSQifrXwlvRB1gVT79Oex7VGUCmpGZCybGCy3phbaK46UzCz4rXzc/MvJL1V0hZJJ2aXwlvWA9mJtV8jkGYHo6H0+mS9waHJ1oXm7LjnKZhZEdoJCpPA75JswbkjvWxv58UlXSlpbzobOjt2oqQbJN2X/t2QHpekT0i6X9JOSc/r/ON0JuuC6dcIpKlZO6hlmcJUrcF3dx3ghOEK60arM+5rZgruPjKzArRzZvlV4BkRsS0izkwvZ7X5+p8GXjHr2PuAGyPibODG9DbAK4Gz08tlwOVtvseiNTOFPo1AqjcazTkKMF1TODRZ42t37uHl557cPDY9T8GT18ysOO2cWe4ADi/mxSPiJmD/rMOvJVk6g/TvT+WOfyYSNwPrJW1ZzPu2K5sf0K/uo1ojZnQDDaUn+pvu3ceBI1O88jmnNO/zMhdm1gvtFJrrwG3prmv5Gc2LHZJ6ckTsTl9jt6TN6fGtwA9yj9uVHtudf7Kky0gyCc4444xFNiGRFZr7NSy1Vo+ZmUJ64v/KbQ+zeqjMBc/c1Lxvdk3B8xTMrAjtBIUvp5eitfrpO+cnfERcAVwBMDY2dlw/8ZuZQp+6j5LRR7maQnqiv/vRp3jNj5zKSLoyKoAkhsqlXE3BmYKZdV87M5qvOtZjOrRH0pY0S9gC7E2P7wJOzz3uNOCRLr/3DFm/fL8KzbNrClkxGeBVPzy352y4Usp1HzlTMLPuO+aZRdLZkr4g6U5JD2aX43jP64BL0+uXAl/JHb8kHYX0QuBA1s1UlH4PSa3NGpKanehXDZW58JxNcx4/XC1xdCrdmMc1BTMrQDs/N/+cZCRQDbgI+AzwF+28uKSrSYayniNpl6S3AL8N/Lik+4AfT29DMnP6QeB+4E+Bt3fwORal0uchqbV6zOgGyrqPXvqszTO6jjLZCCTwjGYzK0Y7NYXRiLhRkiLie8CHJH0T+OCxnhgRF89z18taPDaAd7TRnq6p9nlIaq0RlHMn981rhxmplvjZsdNbPn44173kTMHMitBOUDgqqQTcJ+mdwMPA5mM8Z0no9+ijeqMxY2jpxjXD3P6hn5x3ZFG+5uCagpkVoZ0zy3uAVcC7gB8F3sR0TWBJy/YkmOprpjDzF/9CQ02HZwQFZwpm1n3tjD76dnr1IPDmYpvTW81MoV9BYVZN4ViymkI5t4iemVk3tbOfwl8zd77AAZL1jz4ZEUeLaFgvlJszmvvVfTQ3U1hItlKq5yiYWVHa6T56kCRL+NP08iSwB3hmenvJ6n+hudFRbSAbneR6gpkVpa09miPigtztv5Z0U0RcIOmOohrWC9mQ1L4tc7HYTMH1BDMrSDs/OTdJai4ylF7fmN6cLKRVPZJlCn0rNC+ypuBls82sKO1kCu8F/kHSAyTrE50JvF3SaqZXO12S+j8kNTo6wWejjzzyyMyK0s7oo+slnQ08iyQo3J0rLv9+kY0rWqXvQ1IblDs4wQ9VXFMws2K1kykQERPAPxfclp6r9ntIaqPT7iPXFMysWCv6J+f0gniDsZ/CsWQ1Ba97ZGZFWdFnl+k9mvu1HWd0dIJ3pmBmRes4KEjaImm4iMb02vQezUmm8L5rd/Kbf31nz95/sTUF77pmZkVZzNnlL4C7Jf1etxvTa+VZezTf8i/7uf2RAz17/8XWFKqe0WxmBWmr0JwXES9XsvDOuQW0p6dmF5rHn5rghJGO/0kWrd5pTSHdY8HdR2ZWlLYyBUkvlvTm9PpGYFtELOnZzJAtLJd04xyerHFwosbRqXrP3r/WiI6Glw57SKqZFayd7Tg/CPwG8P700BDw34tsVC9VSyWm6sG+p5LJ2Ud6GhQaixp95AXxzKwo7fSVvA44H/gOQEQ8IumExb6hpHOAa3KHzgL+K7AeeCswnh7/QERcv9j3aVelLGr1BuMHk/l42R7IvdBpTcGT18ysaO0EhcmICEkBkC5vsWgRcQ9wXvpaZZKd3L5EslfDxyOipwXsSknUGsH4UxMAHJ3sTabQaAQRna1j5O4jMytaO2eXz0v6JLBe0luBv6N7S2a/DHgg3fu5L6rlElP1xnRQqPUmKEylE+Y6KRp7noKZFe2YQSH95f4F4FrgHOC/RsT/36X3fwNwde72OyXtlHSlpA2tniDpMknbJW0fHx9v9ZCOJN1Hwd40KEzVozlvoUj1dBjsokYfeUazmRVkwbOLpLKkv4uIGyLiP0fEr0XEDd14Y0lDwGuAv0oPXQ48naRraTfw0VbPi4grImIsIsY2bdp03O2olEpMNaYzBYCjteKDQjY3oqOaQnOTHWcKZlaMBYNCRNSBw5LWFfDerwS+ExF70vfaExH1iGiQdE+9oID3nKOaZgr5oHCkB3WFer3zoOBNdsysaO0Umo8C35V0A3AoOxgR7zrO976YXNeRpC0RsTu9+Trg9uN8/baUS6LeCMYP5jKFHgxLzWoK5UXMU3D3kZkVpZ2g8LfppWskrQJ+HPil3OGPSDoPCOChWfcVJl9oHq2WOTJV70lQqC+i+6i5SqozBTMrSDub7FyV9v8/Mz10T0RMHc+bRsRh4KRZx950PK+5WJWymKo32HdwgjM3rubePQd7MoGttojuI89TMLOitTOj+ULgPuCPgD8G7pV0QcHt6plKqcS+g5NM1YMzTlwF9GYCWzNTWNSQVAcFMytGO91HHwV+Ip10hqRnktQCfrTIhvVKtSx2PX4YgNPToNCTTCGrKXQ4ee0nzj2Zsae1HK1rZnbc2gkK1SwgAETEvZKqBbapp7JMAeD0DVmm0Iug0Hn3kSSuuGSsqCaZmbUVFLZL+hTJPgoAbwR2FNek3sp330x3Hw1mTcHMrGjtBIVfBt4BvAsQcBNJbWFZyBdtT+9lUFhETcHMrGjtBIUK8AcR8TFoLmK3LLbjhOlf6iPVEhvXDAE9mry2iJqCmVnR2jkj3QiM5m6PkiyKtyxkmcKmE4ZZNZTEyJ4sc5F2H3lrTTMbJO0EhZGIOJjdSK+vKq5JvZUtSLdpzXBzyGdvMoXOF8QzMytaO0HhkKTnZTck/ShwpLgm9VbWp7/5hBFKJTFcKfVk+ewp1xTMbAC1U1N4D/BXkh5Jb28BXl9ck3qrWpruPgIYqZZ7stGOawpmNojaWebi25KeRbKXgoC7j3eZi0GS/VLPgsJotdyTGc0ekmpmg2jen6mSni/pFIA0CDwP+H+Bj0o6sUftK1y+0AzJKKTezGh295GZDZ6F+i4+CUwCpGsd/TbwGeAAcEXxTeuNSq7QDGn30YDOaDYzK9pC3UfliNifXn89cEVEXAtcK+m24pvWG5U5mUK5J5lCVlPw3ghmNkgWOiOVJWVB42XA3+fua6dAvSRUW9QUJnpYU/CQVDMbJAud3K8GviFpH8kQ1G8CSHoGSRfSsrButMpotczGNdM1hWyBvCK5pmBmg2jeoBARvyXpRpIhqF+LiEjvKgG/0ovG9cIbf+xpvPRZm5sb2IwO9bam4EzBzAbJgt1AEXFzi2P3duONJT0EPAXUgVpEjKWjmq4BtpFsyflzEfF4N95vPqNDZc7atKZ5e6TSo5pCPemiqrqmYGYDpN9npIsi4ryIyDYJeB9wY0ScTbLm0vt63aCRoR7NU8gyBXcfmdkA6XdQmO21wFXp9auAn+p1A0YqHpJqZitXP4NCAF+TtEPSZemxkyNiN0D6d/PsJ0m6TNJ2SdvHx8e73qjRoVJPgoIXxDOzQdTPoaUviohHJG0GbpB0dztPiogrSCfPjY2NxTEe3rGRSplaI5iqN2ZswNNt00tnD1qyZmYrWd/OSBHxSPp3L/Al4AXAHklbANK/e3vdrtGhMlD87mu1RgMJSs4UzGyA9CUoSFot6YTsOvATwO3AdcCl6cMuBb7S67YNV5OgUPQIpFojXE8ws4HTr+6jk4EvScra8JcR8VVJ3wY+L+ktwPeBn+11w0bToFD0rOZ6I7zEhZkNnL4EhYh4EPiRFscfI1lSo29Gqunua0VnCnVnCmY2ePxTdZYsU+hFTcFzFMxs0DgozDKS1RQK3n3NNQUzG0QOCrNkQeFordiawlNHa6wZXjaLzZrZMuGgMEuzplBwpvDYwQlOSldmNTMbFA4KszRHH9WKDgqTnLR6qND3MDPrlIPCLL2qKTx2yJmCmQ0eB4VZRnswea3RCPYfmmTjGmcKZjZYHBRmaRaaC5y89sSRKRoBJ7r7yMwGjIPCLMOV4ievPXZwAsDdR2Y2cBwUZimVxHClxMRxBoWb7h3npntbL+2d7QG90ZmCmQ0YD5RvYXTo+Lfk/N3/eQ8lwQXP3DTnvv2HkqDgTMHMBo0zhRZGq613X7vn0af4+7v3tPUauw8cZc+TEy3ve+xQ1n3kTMHMBouDQgsj1TJHWhSa//Dr9/Nrf7XzmM+frDXYd3CC8YMTzR3W8vYdnESCDascFMxssDgotDAyT6aw+4kj7D80yeHJ2oLP3/PkUSBZHjvLCvIeOzjBhlVD3orTzAaOg0ILI9XW+zTvPpCc7B954siCz380DQoAe1t0Ie0/5NnMZjaYHBRaaFVTqDeimQHsenzhoJAFD5jOGvIeOzjpeoKZDaSeBwVJp0v6uqS7JN0h6d3p8Q9JeljSbenlVb1uWyapKcwMCo8dnKCW1gcePlamcGD6/lbF5n2HJvVrWqYAAAwySURBVDhptUcemdng6ceQ1Brw3oj4TrpP8w5JN6T3fTwifq8PbZohyRRmFprzv/6P1X30yBNHk9eo1Z0pmNmS0vOgEBG7gd3p9ack3QVs7XU7FjJcLc1ZEC8fFB4+RvfRoweOsnXDKE8cnmLvUzODwlS9wYEjU84UzGwg9bWmIGkbcD5wS3ronZJ2SrpS0oZ5nnOZpO2Sto+Pt54xfLxGq+U5S2dnXULP2LzmmN1Hu588ypZ1I5y8dnhO99HjzYlrzhTMbPD0LShIWgNcC7wnIp4ELgeeDpxHkkl8tNXzIuKKiBiLiLFNm+bOFu6GkWp5bqbw5FGGyiV+eOs6HnlibpdQ3qMHjnDK2hFOXjsyp/uoucSFg4KZDaC+BAVJVZKA8NmI+CJAROyJiHpENIA/BV7Qj7ZBWlOoNYiYnnj26IGjnLJuhNM2jPLok0ep1VuvojpVb7D3qYl5M4Vs3sKJ7j4yswHUj9FHAj4F3BURH8sd35J72OuA23vdtsxItUS9EUzVp4PC7ieSoLB1/Sj1RsyYi5A3/tQEEbBl/SibTxjhsUMTTOUCyGMH3X1kZoOrH6OPXgS8CfiupNvSYx8ALpZ0HhDAQ8Av9aFtQG5PhVqdoXQp7d1PHuF5Z2zg1PWjQFJsPm3DqjnPzQrSp6wbIQIiYN/BCbasS5732KFshVRnCmY2ePox+ugfgFbrO1zf67bMpxkUJuusHanSaAR7DiQn9q0bkpP7IwdaF5sfTYPClnUjNNJ5DXuezAWFgxNUSmLtqBeoNbPB4zNTC6Ozdl/bf3iSyXqDLWn3Ecw/LHV3Giy2rB2lVs+CwnRXUzZHIelFMzMbLA4KLYzM2qf50VyX0Ei1zMY1Q/MOS919IJm4tna0wuZ60kW0Nx8UDk24yGxmA8trH7UwOpT8s2TrH+3OdQkBnLp+lIfnGZb66IGjbFk/giROWj1MuaQZI5D2HZz0cFQzG1gOCi2MVGZmClmX0ClpUNi6fpSHHz/c8rm7DxxpBo9ySWxaMzyj+8grpJrZIHNQaGFkKKspTGcKlZKaI4a2rh/l4SeOzJjHkHn0wFFOWTvavH3y2mH2PDWdKTx2cMLbcJrZwHJQaCHLFI7magonrx2hlG6Ks3XDKEenGs29ljP1RrAnnbiW2bx2pFlTODJZ59Bk3XMUzGxgOSi0MDo0c/TR7gNHOHX99Ik+m6swe7mLfen2m6fkgkIyqzl5XHNvZncfmdmAclBoYaSa/LPkRx+dsm66S6g5LPWJmXWFrCCdDyAnnzDC44enmKjVm5mFV0g1s0HloNDCqmoyUvf7+w8TEew+cHRGl9Bp6QS22TuwZSupzqwpJM/b++QEd+9+CvASF2Y2uDxPoYV1q6q8/Nkn88lvPMBZG1czUWtwytrpoLButMrqoTJf/M7DbF0/ykXP2sy3H9rPVf/4PYBZNYUkK/i7u/bwka/ew3O2ruWHTl3X2w9kZtYmB4V5fOLi83jDFTfz69fuBGae6CXx3p84h0/e9AC//NnvUC6JeiNYN1rlly98OhtyNYMsU/jNv7mTretHufIXnt9cT8nMbNA4KMxj1VCFT136fH768v/ND/YfmVE8BvjFF5/JJf/qaXzz/n18455xxrZt4OXPPrk5GzqTBYW1I1U+/eYXsPmEma9jZjZIHBQWsOmEYT7ziz/GX3zre5x76to591fKJS46ZzMXnbN53tfYsKrKr7z0Gbzs2SfzjM1rimyumdlxU6sJWEvF2NhYbN++vd/NMDNbUiTtiIixVve5c9vMzJocFMzMrMlBwczMmgYuKEh6haR7JN0v6X39bo+Z2UoyUEFBUhn4I+CVwLkk+zaf299WmZmtHAMVFIAXAPdHxIMRMQl8Dnhtn9tkZrZiDFpQ2Ar8IHd7V3qsSdJlkrZL2j4+Pt7TxpmZLXeDFhRa7WY/YyJFRFwREWMRMbZp06YeNcvMbGUYtBnNu4DTc7dPAx6Z78E7duzYJ+l7Hb7HRmDfIto26Jbr54Ll+9n8uZae5fLZnjbfHQM1o1lSBbgXeBnwMPBt4D9ExB1dfI/t883kW8qW6+eC5fvZ/LmWnuX82TIDlSlERE3SO4H/CZSBK7sZEMzMbGEDFRQAIuJ64Pp+t8PMbCUatEJzL1zR7wYUZLl+Lli+n82fa+lZzp8NGLCagpmZ9ddKzBTMzGweDgpmZta0YoLCclpoT9Lpkr4u6S5Jd0h6d3r8REk3SLov/buh321dDEllSbdK+pv09pmSbkk/1zWSho71GoNG0npJX5B0d/q9/atl9H39p/S/w9slXS1pZCl+Z5KulLRX0u25Yy2/IyU+kZ5Pdkp6Xv9a3l0rIigsw4X2asB7I+LZwAuBd6Sf533AjRFxNnBjenspejdwV+727wAfTz/X48Bb+tKq4/MHwFcj4lnAj5B8viX/fUnaCrwLGIuI55AMJX8DS/M7+zTwilnH5vuOXgmcnV4uAy7vURsLtyKCAstsob2I2B0R30mvP0VygtlK8pmuSh92FfBT/Wnh4kk6Dfi3wJ+ltwW8FPhC+pAl97kkrQUuAD4FEBGTEfEEy+D7SlWA0XTy6SpgN0vwO4uIm4D9sw7P9x29FvhMJG4G1kva0puWFmulBIVjLrS3VEnaBpwP3AKcHBG7IQkcwOb+tWzRfh/4daCR3j4JeCIiauntpfjdnQWMA3+edov9maTVLIPvKyIeBn4P+D5JMDgA7GDpf2eZ+b6jZXtOWSlB4ZgL7S1FktYA1wLviYgn+92e4yXp1cDeiNiRP9zioUvtu6sAzwMuj4jzgUMswa6iVtI+9tcCZwKnAqtJulZmW2rf2bEsh/8uW1opQaGjhfaWAklVkoDw2Yj4Ynp4T5bCpn/39qt9i/Qi4DWSHiLp4nspSeawPu2agKX53e0CdkXELentL5AEiaX+fQG8HPiXiBiPiCngi8C/Zul/Z5n5vqNld07JrJSg8G3g7HRExBBJIey6Prdp0dJ+9k8Bd0XEx3J3XQdcml6/FPhKr9t2PCLi/RFxWkRsI/mO/j4i3gh8HfiZ9GFL8XM9CvxA0jnpoZcBd7LEv6/U94EXSlqV/neZfbYl/Z3lzPcdXQdcko5CeiFwIOtmWupWzIxmSa8i+dWZLbT3W31u0qJJejHwTeC7TPe9f4CkrvB54AyS/1l/NiJmF86WBEkXAr8WEa+WdBZJ5nAicCvw8xEx0c/2dUrSeSTF8yHgQeDNJD/Klvz3JenDwOtJRsXdCvxHkv71JfWdSboauJBkeew9wAeBL9PiO0oD4B+SjFY6DLw5Irb3o93dtmKCgpmZHdtK6T4yM7M2OCiYmVmTg4KZmTU5KJiZWZODgpmZNTko2IokqS7pttxlwRnGkt4m6ZIuvO9DkjYu4nk/KelDkjZI8na1VpiB26PZrEeORMR57T44Iv6kyMa04SUkE8IuAP53n9tiy5iDgllOusTGNcBF6aH/EBH3S/oQcDAifk/Su4C3kUzWujMi3iDpROBKksXvDgOXRcROSScBVwObgH8it2aOpJ8nWXZ6iGTi4dsjoj6rPa8H3p++7muBk4EnJf1YRLymiH8DW9ncfWQr1eis7qPX5+57MiJeQDJj9fdbPPd9wPkR8VyS4ADwYeDW9NgHgM+kxz8I/EO6EN51JDNjkfRsklnAL0ozljrwxtlvFBHXkKyTdHtE/DBwe/reDghWCGcKtlIt1H10de7vx1vcvxP4rKQvkyyDAPBi4N8DRMTfSzpJ0jqS7p6fTo//raTH08e/DPhR4NvJigmMMv+CeGcDD6TXV6V7aJgVwkHBbK6Y53rm35Kc7F8D/N+SfoiFl1Ju9RoCroqI9y/UEEnbSdbiqUi6E9gi6TbgVyLimwt/DLPOufvIbK7X5/5+K3+HpBJwekR8nWQzoPXAGuAm0u6fdDG/fekeF/njrwSyfZhvBH5G0ub0vhMlPW12QyJiDPhbknrCR4D/EhHnOSBYUZwp2Eo1mv7iznw1IrJhqcOSbiH50XTxrOeVgf+edg2JZB/iJ9JC9J9L2klSaM6WW/4wcLWk7wDfIFlpk4i4U9L/BXwtDTRTwDuA77Vo6/NICtJvBz7W4n6zrvEqqWY56eijsYjY1++2mPWDu4/MzKzJmYKZmTU5UzAzsyYHBTMza3JQMDOzJgcFMzNrclAwM7Om/wOORhhgXFLGlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score, gamma=0.99')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxcZZ3v8c+3l3RnI4SkgZCFoAYVFxZbQPEKqIOgIt7RGcANGZ2MDm6jV0dHRxBn5qozOo5XRVHiNor7EkYEEfdBmCSyyCIQUSAmQDaydaeru/p3/zinOkVTXX2quk5XV+f7fr3q1XW2Ok9ZeH55nt+zKCIwMzMbra3ZBTAzs6nJAcLMzCpygDAzs4ocIMzMrCIHCDMzq6ij2QVopIULF8by5cubXQwzs5axbt26LRHRU+nYtAoQy5cvZ+3atc0uhplZy5B071jH3MRkZmYVOUCYmVlFDhBmZlaRA4SZmVXkAGFmZhXlFiAkLZX0U0l3SLpN0lsqnCNJH5e0XtItko4rO3aepLvT13l5ldPMzCrLs5vrEPD2iPiNpLnAOknXRMTtZeecAaxIXycAlwAnSDoIuBDoBSK9dnVEbM+xvGZmVia3ABERm4BN6ftdku4AFgPlAeIs4EuRzDl+vaQDJS0CTgGuiYhtAJKuAU4HLs+rvHn608P9fHPt/QwPjz21+vzZM3jNM5cjaWTf1bc9wG1/2lH1s09+fA9PO/yghpXVzKxkUgbKSVoOHAvcMOrQYuD+su0N6b6x9lf67JXASoBly5Y1pLyN9vU19/Pxa++m7Nn/CKUlOZ59ZA+P7Zkzsv893/0tW3YXql635o/buXzliQ0usZnZJAQISXOAbwNvjYidow9XuCSq7H/0zohLgUsBent7p+TqR30DQ8ya0c7tF59e8fjVtz3A33x5HXsHi4/Y318o8rpnHcF7X3RUxeteddkN7B4Yanh5zcwg515MkjpJgsNXIuI7FU7ZACwt214CbKyyvyX1DxaZ2dk+5vEZHcnPUBgafsT+QnF45FjF69rbHnWNmVmj5NmLScBlwB0R8dExTlsNvDrtzXQisCPNXVwNnCZpvqT5wGnpvpbUP1iku0qA6Gp/dIAYHg4Gi1E9QHQ4QJhZfvJsYjoJeBXwW0k3pfv+AVgGEBGfBq4EXgCsB/qA89Nj2yR9AFiTXndxKWHdigYGh5k5I0MNorjvYV96P26AKDpAmFk+8uzF9Csq5xLKzwnggjGOrQJW5VC0SZfUIKo/6OGRNYiRANHuJiYzaw6PpJ4E/YXacxCl911uYjKzJnGAmATj5SBKtYRHNDENZWxicoAws5w4QEyCvRl7MQ1UqEGMFyAGnIMws5w4QEyCvYPFbEnqijmI6r2fCkPDREzJ4R9m1uIcICZB/2CR7o5qD/rkWKUcxHg1CIDBogOEmTWeA8Qk6C9krEGUNRcN1BAg3NXVzPLgADEJ9g4OV09SV+nFNF4319HXmZk1igNEzorDQaE4XDVJ3d4m2ttUOQdRtQbx6KYpM7NGcYDIWWkCvmoD5SAd9Fahm+t44yDKzzUzayQHiJz1pwGiWg4C0i6rZbO5DgwVR/ZXuwagUCyOeY6ZWb0cIHLWXyjVIMYPEBUHymXIQQy4BmFmOXCAyFmpJlAtBwHJw77WgXJdbmIysxw5QOSsv5A8vMcLEF2jps3IOpsrOECYWT4cIHLWP1hDE1OdA+U8DsLM8uAAkbN9SepxejGNykEM1JKDGHSAMLPGc4DI2d6sNYj2MWoQ1QKEaxBmlqPcFgyStAp4EfBQRDy5wvF3AK8oK8cTgZ50Nbk/AruAIjAUEb15lTNvpQAxbpK6Qg6is120tY295pJzEGaWpzxrEF8ATh/rYET8a0QcExHHAO8Gfj5qWdFT0+MtGxxgYt1cq9UewFNtmFm+cgsQEfELIOs60ucCl+dVlmbqz1qDqNDEVC1BDfu6uXpNCDPLQ9NzEJJmkdQ0vl22O4AfSVonaeU416+UtFbS2s2bN+dZ1LrsTRPIWUZS1x4gPBeTmeWn6QECOBP471HNSydFxHHAGcAFkp491sURcWlE9EZEb09PT95lrVmpBlFtTiVIp9oYlYMYL0A4B2FmeZoKAeIcRjUvRcTG9O9DwHeB45tQroYoLTcqjZ1shnSgXK05CAcIM8tRUwOEpHnAycD3y/bNljS39B44Dbi1OSWcuP5CcdyZXOHROYiBoeGR6bzHMjJNuCfrM7Mc5NnN9XLgFGChpA3AhUAnQER8Oj3tfwM/iog9ZZceAnw3/Rd3B/DViLgqr3LmrVSDGE+lbq7jNTHBowOLmVmj5BYgIuLcDOd8gaQ7bPm+e4Cj8ynV5OsfLNI9ToIaKnVzLdI1ThPTyHUOEGaWg6mQg5jWMtcg2tspDgfF4QCy9WKCRwcWM7NGcYDIWf9gcdxBcgBdnY9MOBeKw+P2fIJHTxNuZtYoDhA56y9krUGMChAZaxCjpwk3M2sUB4ic7R0czlSDmDEyKjrpkVRTE5MDhJnloGqSWtKpwEuBpcAQcDfwuYhYPwllmxb2DhbHHUUNjx7TkGUcROk65yDMLA9jPoEkfRB4NXA9MAjcA/we+Kakv5ic4rW+/sEiMzOMgxi9fKi7uZpZs1WrQbwwIp4CIOlrJLOtvkPSt4BfAt+cjAK2uqxJ6pEcRFobGHATk5k1WbUn0LCkg9L3hwHtABGxHag+b4SNqGWgHNSepHYTk5nlpVoN4l+AGyXdCTwBeAOApB7g5kkoW8sbHo6ak9SFoWEiIunmmiUH4SYmM8vJmAEiIr4u6RrgMcD6iHg43b8ZePkkla+llcYnZEpSl3VzHRoOInATk5k1VdUnUNkU3KdKOlPSEyahTNNGaarv7owPekiCSimwZA0QHihnZnkYswYh6WTgI8DDwNOA/wbmSxoEXhUR909OEVvXyHrUNXRzHRgaHqkRZOnmOnqacDOzRqn2BPoYcEZEPA84DhiMiJOAfwYum4zCtbqRGkSWqTY69vViGgkQ40z3Dc5BmFl+qgWI9jTfAHAfcDhARFwDLM67YNNBfyHbetSQTNYHSQ6iUGMTkwOEmeWhWi+mtZIuA64FzgJ+BiNrSI//xLORJqZaezGVFgByN1cza6ZqT6C/AdYBzwR+DLwj3R/A83Mu17Swd7CGXkwjAaK4L0mdqZvrI6cJNzNrlDGfQBExGBGfiog3RsRnI6KY7u+PiHvH+2BJqyQ9JKnicqGSTpG0Q9JN6et9ZcdOl3SnpPWS3lXPF5sKSjmImgbKleUgMk337XWpzSwndc3mKumiDKd9ATh9nHN+GRHHpK+L089uBz4JnAEcBZwr6ah6ytlstSSpy8dB1JqDKF1nZtZI9U73vW68EyLiF8C28c6r4HiSgXn3REQB+BpJDqTl7C1k7+ba2Z7MXpLkIGoPEKVpws3MGqWuABERVzTo/s+QdLOkH0p6UrpvMVA+xmIDVXpNSVopaa2ktZs3bx7rtKbYO5R9oJykZNBbsfZxEOAahJk1XrXpvjsk/Y2kqyTdUvYgf72kzgbc+zfA4RFxNPD/gO+Vbl3h3DEzsBFxaUT0RkRvT09PA4rVOP011CAAutIxDbU0MTlAmFleqnVz/TLJKOqLSP4VD7AEOA/4T+Dsidw4InaWvb9S0qckLUzvtbTs1CXAxoncq1n2TbWRLUCUxjTU1MQ0appwM7NGqRYgjouIx4/atwG4XtJdE72xpEOBByMiJB1PUpvZShKUVkg6AvgTcA4tOjlg/2CRro422tqyzY5eChA1dXN1DcLMclItQGxPV477dkQMA0hqA/4C2D7eB0u6HDgFWChpA3Ah0AkQEZ8GXga8QdIQ0A+cExEBDEl6I3A1yYC8VRFxW53fr6n2FrItFlRSGvRWTzdXT9hnZo1WLUCcA3wI+JSkUkA4EPhpeqyqiDh3nOOfAD4xxrErgSvHu8dUt3dwONMYiJIZdeQgyrvHmpk1UrX1IP5ImmeQtABQRGyZpHJNC/2DxcwJaqgzB+EmJjPLSdZurj0RscXrQdQm63rUJaObmGrJQbiJycwaLWuA+Oqov5ZBsh519qEmM9rbRtaDaBN01DIOwr2YzKzBah0ol607jgFJgKi5BpE2MWVpXoJHThNuZtZI9U61YRn0DxZrSlJ3dexLUmdpXgLnIMwsPw4QOeovFOmuNUldTMZBZFlNrnQNJNOEm5k1Uq0BwosO1GAi3VyzjIGAR04TbmbWSFkDhEb9tQySHEQNSeq6chBuYjKzfGR9ev2vUX8tg1pzEPu6uRYz5yDKpwk3M2ukTE+hiNhd/tfGFxG1B4j29n1J6ow1iPJpws3MGmncp5CkEyWtkbRbUkFSUdLO8a7b3w0MDRNBTUnqrs59TUxZcxCwb5pwM7NGyvIU+gRwLnA3MBN4Hcn6DVbFwGDywK45SV0cZu9g9hoE7MtdmJk1UtYmpvVAe0QUI+LzwKn5Fqv11bIedUkpKOwZGHKAMLOmqzaba0mfpBnATZI+DGwCZudbrNZXChC1DpQD2LV3iGUH1RggnIMwswbL8hR6Fcm6DG8E9pCs9vbSPAs1HZSWG62nBrG71hqEcxBmloNxaxARcW/6th94f9YPlrQKeBHwUEQ8ucLxVwB/n27uBt4QETenx/4I7AKKwFBE9Ga971Sxr4mptgc91BEg3MRkZjnI0ovpRZJulLRN0k5JuzL2YvoCcHqV438ATo6IpwIfAC4ddfzUiDimFYMD7KtBzJqRpRUvUQoKxeGoqReTm5jMLA9Znl4fA/4c+G26JGgmEfELScurHL+ubPN6YEnWz24FfYUhAGbVOBfTyPuMA+VK53o9CDNrtCxPofuBW2sJDnV4LfDDsu0AfiRpnaSV1S6UtFLSWklrN2/enGMRazOSpK4lQJQFBTcxmVmzZalBvBO4UtLPgYHSzoj4aCMKIOlUkgDxrLLdJ0XERkkHA9dI+l1E/KLS9RFxKWnzVG9v75SZTLBvpImpzhpELQPlOtrY6gBhZg2W5Sn0z0Af0A3MLXtNmKSnAp8DzoqIraX9EbEx/fsQ8F3g+EbcbzKNBIjO2nMQsG8hoCy6OtqdgzCzhsvy9DooIk5r9I0lLQO+A7wqIu4q2z8baIuIXen704CLG33/vPWnOYhampi66qxBuInJzPKQJUD8WNJpEfGjWj5Y0uXAKcBCSRuAC4FOgIj4NPA+YAHwKUmwrzvrIcB3030dwFcj4qpa7j0V9BWKdLSpxvEM+4KJx0GYWbNlCRAXAO+UNAAMkqwJERFxQLWLIuLccY6/jmRep9H77wGOzlCuKa2vUKyp9gD15yDczdXM8pBloFxD8g37m/5CsaYENTwyKHTV0s3VTUxmloNMGdQ0mby8/PyI+E5OZZoW+gaLNQ2SgwnWIBwgzKzBxn2CpVNmPBW4DSg9hYIkwWxj6C8M1TRRH0xgHEQ6TXhEkOZuzMwmLMs/cU+MiKNyL8k00zfBJqaaRlKn1yULDdV2TzOzsWR5Cv1akgNEjepJUtfbzbV0nZuZzKyRstQgvkgSJB4gGUld6sX01FxL1uL6C0UOOaCrpmsmMtUGOECYWWNlCRCrSNaE+C37chA2jr7BoZqT1G1toqNNDA1HzTkIwF1dzayhsjzB7ouI1bmXZJrpr6OJCZLawFChWF8OwjUIM2ugLAHid5K+ClzBIyfrcy+mKvoKRWbV2IsJkod9X6FY83oQ4ABhZo2VJUDMJAkM5fMxuZtrFRFB/2DtvZhgX3NRPU1MXhPCzBopy0jq8yejINPJ3sFhImBmjTkI2BcY6klSO0CYWSNlGSjXTbJew5NIpvwGICL+KsdytbR6VpMrGQkQzkGYWZNleQp9GTgUeD7wc5KlQXflWahWV1oLoq4kdR1NTF0d7sVkZo2X5Sn0uIj4R2BPRHwReCHwlHyL1dpKy43WU4PoqqeJKZ0m3DUIM2ukLE+hwfTvw5KeDMwjmbjPxlDPcqMlbmIys6kiSxb1UknzgX8EVgNzSBb7sTGUchAza1hutGRGRxsz2ttqmnRv31xMxZrvZ2Y2lnH/mRoRn4uI7RHx84h4TEQcnK4INy5JqyQ9JOnWMY5L0sclrZd0i6Tjyo6dJ+nu9HVe9q/UfP0TqUG0t9XUvASuQZhZPrL0Ynpbhd07gHURcdM4l38B+ATwpTGOnwGsSF8nAJcAJ0g6iGSJ0l6SMRfrJK2OiO3jlXcqmGgTU80Bot0BwswaL8uTqBd4PbA4fa0kWWv6s5LeWe3CiPgFsK3KKWcBX4rE9cCBkhaR9Ji6JiK2pUHhGuD0DGWdEvon0oupo72m/ENyjcdBmFnjZWkkXwAcFxG7ASRdCHwLeDawDvjwBO6/GLi/bHsD+wJRpf2PImklSdBi2bJlEyhK4+wbB1F7DuLkI3tYOGdGTdeUFibaO+gchJk1TpZ/qi4DCmXbg8DhEdFP2dxMdaqUiY0q+x+9M+LSiOiNiN6enp4JFqcx+ibQzfVlT1vChWc+qaZrSs1SuwccIMyscbL8E/erwPWSvp9unwlcLmk2cPsE778BWFq2vQTYmO4/ZdT+n03wXpOmv1BEoqYJ9yZqblcHuwcGxz/RzCyjLL2YPgD8NfAwSXL69RFxcUTsiYhXTPD+q4FXp72ZTgR2RMQm4GrgNEnz0y62p6X7WkJpJtfJXB96TncHu/cOTdr9zGz6y9RIHhHrSPINNZF0OUlNYKGkDSQ9kzrTz/w0cCXwAmA90Aecnx7bJukDwJr0oy6OiGrJ7iklWW609vzDRMzt7mCXA4SZNVCuT7GIOHec4wFcMMaxVSSr2bWc/sJQXfmHiZjT1cGuAQcIM2ucyWsk34/0FepbC2Ii5nR1uonJzBrKASIH/YP1LTc6EXO7O9jtGoSZNVDNAULSjyX9UNKL8ijQdNCcGoQDhJk1Vj05iFcDi4ATG1yWaaOvUGT+rNoGu02UezGZWaNlChCSZgLLIuLOiNhIMlah5l5N+4v+wtCkNzHN6eqgUBxmYKhIV8fk3tvMpqdxm5gknQncBFyVbh8jaXXeBWtlpXEQk2ludxLrXYsws0bJkoO4CDieZKAc6Qyuy/MrUuvrL0x+knpOVxognIcwswbJEiCGImJH7iWZJiKCvsHmJKkBD5Yzs4bJkoO4VdLLgXZJK4A3A9flW6zWVSgOUxyOyQ8Q3a5BmFljZalBvAl4EsnMrZcDO4G35lmoVrZvLYhJnmqjqxNwDsLMGmfcp1hE9AHvSV82jomsJjcRpRrELs/oamYNMmaAkHQFY6zBABARL86lRC2uaQGiy72YzKyxqtUg/m3SSjGNjDQxNambqyfsM7NGGTNARMTPJ7Mg08VElhudiK6ONjrb5RqEmTXMuE+xtOfS/wWOArpL+yPiMTmWq2WVlhud7HEQkjwfk5k1VJZeTJ8HLgGGgFOBLwFfzrNQray/STkI8HxMZtZYWQLEzIi4FlBE3BsRFwHPyfLhkk6XdKek9ZLeVeH4v0u6KX3dJenhsmPFsmMtM7VHs5LUkKwJ4RyEmTVKlobyvZLagLslvRH4E3DweBdJagc+CfwZsAFYI2l1RNxeOici/q7s/DcBx5Z9RH9EHJPta0wd/WkOYrKbmADmdrkGYWaNk6UG8VZgFskI6qcBrwTOy3Dd8cD6iLgnIgrA14Czqpx/LslAvJa2rwYxuUlqSJuYXIMwswbJMlBuTfp2N3B+DZ+9GLi/bHsDcEKlEyUdDhwB/KRsd7ektSS5jw9GxPfGuHYlsBJg2bJlNRQvH31N6uYKyViIP2zZM+n3NbPpKct039dIOrBse76kqzN8tirsG2vg3TnAtyKiWLZvWUT0Ai8HPibpsZUujIhLI6I3Inp7enoyFCtf/YNFujraaG+r9PXzNae7w5P1mVnDZGliWhgRI8njiNhOhhwESY1hadn2EpKFhio5h1HNS+nCRETEPcDPeGR+YsrqKww1JUENSQ5i115PtWFmjZElQAxLGmm7SZuDxpyCo8waYIWkIyTNIAkCj+qNJOnxwHzg12X75kvqSt8vBE4Cbh997VSUrEc9+fkHSJqYBoaGKQwNN+X+Zja9ZHmSvQf4laTSyOpnk7b5VxMRQ2mvp6uBdmBVRNwm6WJgbUSUgsW5wNciojzoPBH4jKRhkiD2wfLeT1NZMxYLKilN2LdnYIgZHZO7JraZTT9ZktRXSToOODHd9XcRsSXLh0fElcCVo/a9b9T2RRWuuw54SpZ7TDVJDaJJAaJsVbn5sx0gzGxixmxiknS4pHkAaUDYQzKm4dVpk5FV0F8oNqUHE5RN2OdEtZk1QLUcxDeA2QCSjgG+CdwHHA18Kv+itaa+weYlqeeUFg3yWAgza4BqTUwzSz2JSAbHrYqIj6Sjqm/Kv2itqZlJ6rkjy466J5OZTVy1GkR5R/7nANcCRIS7yFQxFZLUbmIys0ao9k/dn0j6BrCJpBvqTwAkLQIKk1C2ltTMJPXcsiS1mdlEVQsQbwXOBhYBz4qIUrvFoXh96oqGisPs6B/kwFnNyeGXahCesM/MGqHainJBMsHe6P035lqiFra9L4mhC+c0J0DM7GynTa5BmFljZBlJbRlt3TMAwILZXU25f2lVOecgzKwRHCAaaOvuJDWzoEk1CIC53Z0OEGbWEJkChKSZ6ZxJVsWW3UkNollNTEC6LrW7uZrZxGWZ7vtMknEPV6Xbx7TSEqCTaaQG0aQmJvCiQWbWOFlqEBeRrA73MEBE3AQsz69IrWvrngHa28S8mZ1NK8McLztqZg2SJUAMRcSO3EsyDWzdXeCg2TNoa8JiQSVzujvY5RqEmTVAljkhbpX0cqBd0gqStamvy7dYrWnL7gILmjyL6lzXIMysQbLUIN4EPAkYIFn1bSfJIDobZeueARbOaV7+AUpJagcIM5u4cQNERPRFxHsi4unACcCHImJvlg+XdLqkOyWtl/SuCsdfI2mzpJvS1+vKjp0n6e70dV4tX6pZtu4uNLWLKyRNTH2FIsXhLIv+mZmNLUsvpq9KOkDSbOA24E5J78hwXTvwSeAM4CjgXElHVTj16xFxTPr6XHrtQcCFJAHpeOBCSfMzf6sm2bp7oKk9mOCRiwaZmU1EliamoyJiJ/ASktXhlgGvynDd8cD6iLgnIgok03aclbFczweuiYhtEbEduAY4PeO1TdFfKLKnUGx6DeKAbq8JYWaNkSVAdErqJAkQ308n7cvSfrEYuL9se0O6b7SXSrpF0rckLa3x2imjNM1GMwfJwb4J+3b0ebCcmU1MlgDxGeCPJKvL/ULS4SSJ6vFU6us5OrBcASyPiKcCPwa+WMO1yYnSSklrJa3dvHlzhmLlYyoMkgM45IDk/g/typQmMjMbU5Yk9ccjYnFEvCAS9wKnZvjsDcDSsu0lwMbyEyJia0QMpJufBZ6W9dqyz7g0InojorenpydDsfIxMlFfk2sQh86bCcCmHQ4QZjYxY46DkPTKiPhPSW8b45SPjvPZa4AVko4A/gScA7x81D0WRcSmdPPFwB3p+6uBfylLTJ8GvHuc+zXVlrQG0exurgfP7UJygDCzias2UG52+nduPR8cEUOS3kjysG8nWdP6NkkXA2sjYjXwZkkvBoaAbcBr0mu3SfoASZABuDgittVTjskyFWZyBehsb6NnThcP7OhvajnMrPVVWzDoM+nf99f74RFxJUnPp/J97yt7/27GqBlExCpgVb33nmxbdw8ws7OdWTOyDE7P16J53a5BmNmEZRkHsUTSdyU9JOlBSd+WtGQyCtdKtu5p/iC5kkPndfOAA4SZTVCWXkyfB1YDh5F0Nb0i3WdltuweYEGT8w8li+bNdIAwswnLEiB6IuLzETGUvr4ANK+70BS1dXeBhU2eqK9k0bxudg0MsWuvx0KYWf2yBIgtkl4pqT19vRLYmnfBWs3WPQNTqokJcC3CzCYkS4D4K+AvgQeATcDL0n2Wioh0or6p08QE7upqZhMzbpebiLiPZIyCjWFn/xBDw9H0tSBKFrkGYWYNUG2g3PvGOgZERHwgh/K0pC0j8zBNjRrEwel0G65BmNlEVKtB7KmwbzbwWmAB4ACRmiqD5Eq6OtpZOGcGD+z0YDkzq1+1gXIfKb2XNBd4C3A+ybTdHxnruv3R1t3pPExNnqiv3KJ5M12DMLMJqZqklnSQpH8CbiEJJsdFxN9HxEOTUroWsWVPaR6mqVGDAA+WM7OJGzNASPpXkrmQdgFPiYiL0sV7bJRtaRPT/CmSpIYkUb3xYTcxmVn9qtUg3k4yevq9wEZJO9PXLklZ1oPYb2zdM8CBszrpbM/Sa3hyHDqvm517h9jjleXMrE7VchBT52k3xW3asZeD506d/AOUdXXduZfH9sxpcmnMrBU5CDTA3Q/uYsXBdc2KnptDD0gGyzkPYWb1coCYoP5CkXu39XHkIVMrQJRqEO7JZGb1coCYoPUP7SYCHn/o1GrG2TcfkxPVZlafXAOEpNMl3SlpvaR3VTj+Nkm3S7pF0rWSDi87VpR0U/panWc5J+LOB3cBTLkaRHdnOwfNnuEahJnVLbflzyS1A58E/gzYAKyRtDoibi877UagNyL6JL0B+DBwdnqsPyKOyat8jXLXg7uY0dHG4Qtmj3/yJDv0AK8sZ2b1y7MGcTywPiLuiYgCyQjss8pPiIifRkRfunk90HIr1d35wC5WHDyH9jY1uyiP4qVHzWwi8gwQi4H7y7Y3pPvG8lrgh2Xb3ZLWSrpe0kvGukjSyvS8tZs3b55Yietw14O7ePwUa14qWTJ/Jvdu3UNxOJpdFDNrQXkGiEr/pK74pEoXIeoF/rVs97KI6AVeDnxM0mMrXRsRl0ZEb0T09vRM7kJ3O/oH2bRjL0ceOjUDxNFLD6SvUOTuh3Y1uyhm1oLyDBAbgKVl20uAjaNPkvQ84D3AiyNioLQ/Ijamf+8BfgYcm2NZ63J3mqCeqjWI45bNB+A39z7c5JKYWSvKM0CsAVZIOkLSDOAc4BG9kSQdC3yGJDg8VLZ/vqSu9P1C4CSgPLk9JYz0YJqiNYjDF8xi/qxObrzPU2iZWe1y68UUEUOS3ghcDbQDqyLiNkkXA2sjYjVJk9Ic4JuSAO6LiBcDTwQ+I2mYJIh9cFTvpynhrgd2MX6IsJgAAAvMSURBVKerg8PSMQdTjSSOXTafG+93DcLMapdbgACIiCuBK0fte1/Z++eNcd11wFPyLFsj3PngLo48ZA5pcJuSjl16ID/53UPs6B9k3szOZhfHzFqIR1LXKSK484FdU26A3GjHHZ7kIW5yLcLMauQAUactuwts7xuc8gHiqUvmIeE8hJnVzAGiTneVejBN0QR1ydzuTo48eC433ucahJnVxgGiTtf9fgvtbeKoRQc0uyjjOu7wA7np/ocZ9oA5M6uBA0QdIoIrbt7EMx+7YEotMzqWY5fOZ0f/IPds2dPsophZC3GAqMMtG3Zw37Y+zjz6sGYXJZNjlx0IOA9hZrVxgKjDFTdvpLNdPP9Jhza7KJk8tmcOB3R38Ovfb212UcyshThA1Gh4OPjBbzdx8pEHt8y4grY2cebRh/Ffv93Elt0D419gZoYDRM3W3bedTTv2cubRi5pdlJqcf9IRFIaG+cr19zW7KGbWIhwganTFzRvp7mzjeU88pNlFqcnjDp7DKY/v4cvX38vAULHZxTGzFuAAUYPC0DBX/nYTz33iIczuynWWkly89llHsGX3AFfcvKnZRTGzFuAAUYP/uPYutuwucO7TlzW7KHV51uMWcuQhc7jsV38gwmMizKw6B4iM1t27nUt+9nvO7l3Ks1YsbHZx6iKJvzrpCO7YtJOrbn2g2cUxsynOASKDvsIQb//GTSyaN5P3vuiJzS7OhLzk2MU8ZfE83vaNm7nZE/iZWRUOEOMoDA3znu/eyr3b+vjIXx7N3O7W6No6lu7Odi57TS8L5szgtV9cw31b+5pdJDObohwgqvjDlj289JLr+O6Nf+Itz13BiY9Z0OwiNcTBc7v5wvnHM1gMXnnZDVz3+y3NLpKZTUG5BghJp0u6U9J6Se+qcLxL0tfT4zdIWl527N3p/jslPT/Pco5279Y9fPSau3jhx3/Jfdv6+MyrnsZbn3fkZBYhd487eA6rXvN0isPByz97A3/9pbXctnGHk9dmNiK3vpqS2oFPAn8GbADWSFo9aunQ1wLbI+Jxks4BPgScLekokjWsnwQcBvxY0pER0fAO/MPDwe2bdvK7B3Zxx6adrLt3Ozfd/zASnPr4g/mnlzyZww6c2ejbTglPO3w+1779ZC771R/41E/Xc83tD7JoXjcnH9nDkxfPY/mC2Sw7aBYHzZnB7BntU3rlPDNrvDw78x8PrI+IewAkfQ04CygPEGcBF6XvvwV8QslT6CzgaxExAPxB0vr0837d6EIOR/DSS65jYGiYro42nrDoAP7+9CfwkmMPY9G86RkYynV3tnPBqY/jnKcv5cd3PMjP7tzMD27ZxNfW3P+I8zrbxdzuTro72ujqbKejzcHCbKqYP2sG33j9Mxr+uXkGiMVA+VNmA3DCWOdExJCkHcCCdP/1o65dXOkmklYCKwGWLat9fEJHexuXvrqXxQfO5IiFs2nfTx98C+Z0cfbTl3H205cxPBw8uGsvf9zSx/3b+3i4r8DDfYPs2jvE3sEi/YNFht0UZTZlHJBT55k8A0SlJ+3op8pY52S5NtkZcSlwKUBvb29dT62Tj+yp57Jpq61NLJo3k0XzZvIMpkdi3sxql2eSegOwtGx7CbBxrHMkdQDzgG0ZrzUzsxzlGSDWACskHSFpBknSefWoc1YD56XvXwb8JJJuNKuBc9JeTkcAK4D/ybGsZmY2Sm5NTGlO4Y3A1UA7sCoibpN0MbA2IlYDlwFfTpPQ20iCCOl53yBJaA8BF+TRg8nMzMam6dTvvbe3N9auXdvsYpiZtQxJ6yKit9Ixj6Q2M7OKHCDMzKwiBwgzM6vIAcLMzCqaVklqSZuBe2u4ZCEwHacyna7fC6bvd/P3aj3T5bsdHhEVRwtPqwBRK0lrx8ret7Lp+r1g+n43f6/WM52/W4mbmMzMrCIHCDMzq2h/DxCXNrsAOZmu3wum73fz92o90/m7Aft5DsLMzMa2v9cgzMxsDA4QZmZW0X4bICSdLulOSeslvavZ5amXpKWSfirpDkm3SXpLuv8gSddIujv9O7/ZZa2HpHZJN0r6r3T7CEk3pN/r6+lU8i1H0oGSviXpd+lv94zp8JtJ+rv0v8NbJV0uqbtVfzNJqyQ9JOnWsn0VfyMlPp4+T26RdFzzSt44+2WAkNQOfBI4AzgKOFfSUc0tVd2GgLdHxBOBE4EL0u/yLuDaiFgBXJtut6K3AHeUbX8I+Pf0e20HXtuUUk3cfwBXRcQTgKNJvmNL/2aSFgNvBnoj4skk0/yfQ+v+Zl8ATh+1b6zf6AySdWtWkCyBfMkklTFX+2WAAI4H1kfEPRFRAL4GnNXkMtUlIjZFxG/S97tIHjSLSb7PF9PTvgi8pDklrJ+kJcALgc+l2wKeA3wrPaVVv9cBwLNJ1kMhIgoR8TDT4DcjWWNmZrpC5CxgEy36m0XEL0jWqSk31m90FvClSFwPHChp0eSUND/7a4BYDNxftr0h3dfSJC0HjgVuAA6JiE2QBBHg4OaVrG4fA94JDKfbC4CHI2Io3W7V3+0xwGbg82nz2eckzabFf7OI+BPwb8B9JIFhB7CO6fGblYz1G03LZ8r+GiBUYV9L9/eVNAf4NvDWiNjZ7PJMlKQXAQ9FxLry3RVObcXfrQM4DrgkIo4F9tBizUmVpO3xZwFHAIcBs0maXkZrxd9sPNPlv81H2F8DxAZgadn2EmBjk8oyYZI6SYLDVyLiO+nuB0tV3PTvQ80qX51OAl4s6Y8kTYDPIalRHJg2X0Dr/m4bgA0RcUO6/S2SgNHqv9nzgD9ExOaIGAS+AzyT6fGblYz1G02rZ0rJ/hog1gAr0t4VM0gSaaubXKa6pO3ylwF3RMRHyw6tBs5L358HfH+yyzYREfHuiFgSEctJfp+fRMQrgJ8CL0tPa7nvBRARDwD3S3p8uuu5JOuvt/RvRtK0dKKkWel/l6Xv1fK/WZmxfqPVwKvT3kwnAjtKTVGtbL8dSS3pBST/Im0HVkXEPze5SHWR9Czgl8Bv2ddW/w8keYhvAMtI/o/7FxExOuHWEiSdAvyfiHiRpMeQ1CgOAm4EXhkRA80sXz0kHUOSfJ8B3AOcT/IPtpb+zSS9HzibpHfdjcDrSNriW+43k3Q5cArJtN4PAhcC36PCb5QGxE+Q9HrqA86PiLXNKHcj7bcBwszMqttfm5jMzGwcDhBmZlaRA4SZmVXkAGFmZhU5QJiZWUUOEGaApKKkm8peVUc2S3q9pFc34L5/lLSwjuueL+kiSfMlXTnRcphV0jH+KWb7hf6IOCbryRHx6TwLk8H/IhmA9mzgv5tcFpumHCDMqkin+vg6cGq66+URsV7SRcDuiPg3SW8GXk8yOOz2iDhH0kHAKpKJ+fqAlRFxi6QFwOVAD/A/lM3hI+mVJNNlzyAZ6Pi3EVEcVZ6zgXenn3sWcAiwU9IJEfHiPP43sP2Xm5jMEjNHNTGdXXZsZ0QcTzJS9mMVrn0XcGxEPJUkUAC8H7gx3fcPwJfS/RcCv0on6VtNMiIXSU8kGYF8UlqTKQKvGH2jiPg6ybxNt0bEU4Bb03s7OFjDuQZhlqjWxHR52d9/r3D8FuArkr5HMhUDwLOAlwJExE8kLZA0j6RJ6M/T/T+QtD09/7nA04A1yawNzGTsyfpWAL9P389K1wExazgHCLPxxRjvS15I8uB/MfCPkp5E9emfK32GgC9GxLurFUTSWpK5gTok3Q4sknQT8KaI+GX1r2FWGzcxmY3v7LK/vy4/IKkNWBoRPyVZ3OhAYA7wC9ImonSywS3pOh3l+88ASutOXwu8TNLB6bGDJB0+uiAR0Qv8gCT/8GHgPRFxjIOD5cE1CLPEzPRf4iVXRUSpq2uXpBtI/kF17qjr2oH/TJuPRLL28sNpEvvzkm4hSVKXpoh+P3C5pN8APyeZEZSIuF3Se4EfpUFnELgAuLdCWY8jSWb/LfDRCsfNGsKzuZpVkfZi6o2ILc0ui9lkcxOTmZlV5BqEmZlV5BqEmZlV5ABhZmYVOUCYmVlFDhBmZlaRA4SZmVX0/wGC3LAXMZanZwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax2 = fig.add_subplot(121)\n",
    "plt.plot(np.arange(1, len(arr_noise)+1), arr_noise)\n",
    "plt.ylabel('Noise Scale, gamma=0.99')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Watch a Smart Agent!\n",
    "\n",
    "Each step of the Smart Agent animation includes the cl of the following _action_, _state_ and _reward_.     \n",
    "This also includes a slight _delay_ (0.1 sec) for better observation of the animation, and  further rendering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "timesteps:  200\n",
      "time :  11.815999984741211\n"
     ]
    }
   ],
   "source": [
    "t1 = time.time()\n",
    "\n",
    "state = env.reset()\n",
    "timesteps = 0\n",
    "while True:\n",
    "    action = policy.act(state)\n",
    "    env.render()\n",
    "    time.sleep(0.05)\n",
    "    timesteps += 1\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "\n",
    "print('timesteps: ', timesteps)\n",
    "delta = time.time() - t1\n",
    "print('time : ', delta)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
